DX-64328 Array types for Gandiva #58

lriggs · 2023-10-25T22:09:59Z

Add List input and output types for Gandiva functions. Add new reference implementations for array_contains and array_remove, tested via integration with Dremio. int32, int64, double and float list types have been tested.

Support List types in function specification and llvm code generation.
Pass back function type information through the expression registry.

See 1p here: https://docs.google.com/document/d/1exwXdUUnk5FqZLzVZyTdhqgwxTk0u9bL54aLVNM5Tas/edit

github-actions · 2023-10-25T22:10:21Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/ListVectorExpander.java

java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java

cpp/src/gandiva/annotator.cc

cpp/src/gandiva/array_ops.cc

cpp/src/gandiva/projector.cc

DenisTarasyuk · 2023-11-06T16:15:14Z

cpp/src/gandiva/projector.cc

+    if (arrow::is_binary_like(internal_type->id())) {
+      child_data = arrow::ArrayData::Make(
+          internal_type, 0 /*initialize length*/,
+          {nullptr, std::move(buffers[2]), std::move(buffers[3])}, 0);


0,1,2,3 indexes as constants

cpp/src/gandiva/tests/list_test.cc

DenisTarasyuk · 2023-11-06T16:17:48Z

java/gandiva/src/main/cpp/jni_common.cc

+  }
+
+  if (type->id() == arrow::Type::LIST) {
+    jlong offsets_addr = in_buf_addrs[buf_idx++];


Does gandiva tests perform any memory leak checks? How do we make sure that added code does not result in memory leaks?

DenisTarasyuk · 2023-11-06T16:18:09Z

java/gandiva/src/main/cpp/jni_common.cc

+      child_data = arrow::ArrayData::Make(internal_type, 0,
+                                          {std::move(buffers[2]), std::move(data_buffer)});
+    }
+    if (arrow::is_binary_like(internal_type->id())) {


Is this block required?

…ocator, CallBack) for StructVector (dremio#59)

…ocator, CallBack) for MapVector (dremio#60) * apacheGH-38511: [JAVA] added impl of getTransferPair(Field, BufferAllocator, CallBack) for MapVector

Add List input and output types for Gandiva functions. Add new reference implementations for array_contains and array_remove, tested via integration with Dremio. int32, int64, double and float list types have been tested. Support List types in function specification and llvm code generation. Pass back function type information through the expression registry. See 1p here: https://docs.google.com/document/d/1exwXdUUnk5FqZLzVZyTdhqgwxTk0u9bL54aLVNM5Tas/edit

lriggs and others added 26 commits October 25, 2023 13:48

Fix merge

dececfd

Test geo function return types

e56dba6

Fix unix build

05fb64f

Struct testing

0593acd

Merge in list/array stuff

690f5ea

add two list type input gandiva function

41c5083

Fix leftover merge

aa0a15c

Fix merge

0fdb09b

Somewhat working for int32

1976d7c

Almost working

b2a8380

lists kind of working

0db2c7a

add stuff. 100 rows

89d9f2d

Removed logging and testing bigger size

4249251

Working version with 100 rows and correct data

4832d63

update scripts

bcfcd88

Using some dynamic sizes

a82ef6a

Working with 1million rows

5b38893

Everything working.

2f33cda

Remove some logging.

e29a751

experiement 1

eb0c497

Return validity basically working.

7f62ffb

Working input validiy, but large data is broken again.

7a06e9e

Everything working

a9a7f51

Cleanup, tested

a4ee4ae

Cleanup and test

316b822

Cleanup, tested

7d74608

github-actions bot added Component: C++ Component: C++ - Gandiva Component: Java labels Oct 25, 2023

lriggs added 4 commits November 2, 2023 17:52

cleanup

112d540

Disable tests for now.

0e3bd9e

Remove uneeded changes.

662e9f1

Remove uneeded includes

4bd53e2

DenisTarasyuk reviewed Nov 6, 2023

View reviewed changes

lriggs and others added 11 commits November 6, 2023 13:37

Fix unit tests

b4af98a

Cleanup.

a057f58

Cleanup

be455ab

Tidy up function results with nulls present.

7c7a939

Fix unit test

fb55515

Make listreturn type optional. update unit test.

6639639

Fix javadoc

054ef11

apacheGH-38511: [JAVA] added impl of getTransferPair(Field, BufferAll…

2d9cdbf

…ocator, CallBack) for StructVector (dremio#59)

apacheGH-38511: [JAVA] added impl of getTransferPair(Field, BufferAll…

f477cc8

…ocator, CallBack) for MapVector (dremio#60) * apacheGH-38511: [JAVA] added impl of getTransferPair(Field, BufferAllocator, CallBack) for MapVector

Remove unneeded code.

b7c5288

Return list parameter type information through the function registry.

b6593b3

lriggs changed the base branch from dremio_24.3_12.0 to dremio_25.0_12.0 November 20, 2023 19:12

lriggs changed the title ~~WIP: DX-64328 Array types for Gandiva~~ DX-64328 Array types for Gandiva Nov 20, 2023

lriggs merged commit a225426 into dremio:dremio_25.0_12.0 Nov 20, 2023
22 of 23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DX-64328 Array types for Gandiva #58

DX-64328 Array types for Gandiva #58

lriggs commented Oct 25, 2023 •

edited

Loading

github-actions bot commented Oct 25, 2023

DenisTarasyuk Nov 6, 2023

DenisTarasyuk Nov 6, 2023

DenisTarasyuk Nov 6, 2023

DX-64328 Array types for Gandiva #58

DX-64328 Array types for Gandiva #58

Conversation

lriggs commented Oct 25, 2023 • edited Loading

github-actions bot commented Oct 25, 2023

DenisTarasyuk Nov 6, 2023

Choose a reason for hiding this comment

DenisTarasyuk Nov 6, 2023

Choose a reason for hiding this comment

DenisTarasyuk Nov 6, 2023

Choose a reason for hiding this comment

lriggs commented Oct 25, 2023 •

edited

Loading